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ABSTRACT 

This paper suggests that evaluation research about education 
policy is intended to affect decisions and typically addresses pedagogy, 
politics, and economics, all simultaneously. The paper begins with two 
assumptions: first, the need to do credible science, and second, the need for 
science to advance the contribution that instructional technology might make 
to learning. It is divided into six sections: (1) Evaluation Research about 

Instructional Technology Policies; (2) Pedagogy (efficacy of instructional 
technology, multiple sources of learning, learning outside the school, and 
implications for a research agenda) ; (3) Politics; (4) Economics (capital 
decisions, public benefits of private investment, and roles of government); 

(5) Four Instructional Technology Additions to the Existing Conception of 
Schooling (adding an education focus to the school focus, adding a learning 
focus to the teaching focus, adding homes to schools -adding parents to 
teachers, and moving learning to the learner); and (6) Four Evaluation 
Research Questions from Four Perspectives (science, pedagogy, politics, and 
economics). (Contains 10 footnotes.) (AEF) 
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’ B. Adding a Learning Focus to Our Teaching Focus 

C. Adding Homes to Schools: Adding Parents to Teachers 

D. Moving Learning to the Learner 

VI. FOUR EVALUATION RESEARCH QUESTIONS FROM FOUR PERSPECTIVES 



PREFACE 

[October 22, 1707, The English Channel] 

"Returning home victorious from Gibraltar... [Admiral] Sir Clowdisley... summoned all his 
navigators... The consensus placed the English fleet safely west of... the Brittany peninsula. 

But as the sailors continued north, they discovered to their horror that they had misgauged 
their longitude near the Sicily Isles [which] became unmarked tombstones for two thousand 
of Sir Clowdisley's troops... [and] four of the five warships."! 

Appalled by the loss of lives and ships, in the Longitude Act of 1714, the Parliament promised a prize of 
£20,000 for a better way to navigate than throwing a log overboard and watching it drift off. Without a 
way to measure time accurately, ships could not determine noon and without a way to determine noon, 
they could not determine their east-west location. The lack of measurement had serious consequences. 

I begin with two assumptions: first, that we want to do credible science and second that we want that 
science to advance the contribution that instructional technology might make to learning. Trying to 
advance instructional technology makes my perspective partisan or political in the sense that I am 
acknowledging a particular value interest in the outcomes of this application of science. 

I. EVALUATION RESEARCH ABOUT INSTRUCTIONAL TECHNOLOGY 
POLICIES 

"640K ought to be enough for anyone." Bill Gates, 1981 

Doing research costs money and most evaluation research is paid for by clients— governments or private 
organizations. Adding a client may complicate evaluation in the same way that adding a patron 
complicates art: some clients and some patrons have expectations in addition to the outcomes of the 
otherwise pure event of science or art. The evaluation fish swims in the sea of politics, and should. Anne 
L. Bryant, executive director of the National School Boards Association says, "School Boards are going 
to be asking increasingly: 'Demonstrate to us that [computer-based instruction] has results."^ 

Imagine a senior government official on the cell phone, in a cab, running late oh her way up Constitution 
Avenue to a congressional hearing. She knows she will face pointed questions about "All these 
computers we put in schools". If you answer the call, will you want to help address that skepticism? If 
you are good at helping, you might add seven figures to the appropriations authorization: if you are really 
good, you might add eight figures. If you want to help, you have to compel belief and that is likely to be 
more than good science, it is likely to address pedagogy, politics and economics, all at once. 

In my view, evaluation research about education policy is intended to effect decisions and typically 
addresses pedagogy, politics and economics, all simultaneously. 
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II. PEDAGOGY 

"There is no reason why anyone would want a computer in their home." Ken Olson, 

President, Chair and Founder of Digital Equipment Corp., 1977 

A. The Efficacy of Instructional Technology. The overriding question is how powerful is instructional 
technology? A second order question is, how do we know how powerful that technology is? 

First, I believe that instructional technology works Instructional technology only works for some kids, 
in some topics and under some conditions but that is true of all pedagogy, all systems for teaching or 
learning. There is nothing that works for every purpose, for every learner and all the time. 

Emphasizing the things that instructional technology has not done has its political uses just as surely as 
saying that technology works. Oiie continuity among critics of instructional technology is the idea that all 
teachers are always preferable to all machines, e.g., William L. Rukeyser's statement that, "The best 
teacher has always been a person, not a machine. The sub-title of the cover story of The School 
Administrator for April 1999 was "A Leading psychologist calls for slowing the rush toward computing". 
In that piece, the critic of instructional technology, Jane Healy acknowledges that, "...(W)ell 
implemented simulations and conceptually driven programs may improve learning— if a good teacher is 

in charge."^ 

But what is known about the learning efficacy of such ubiquitous features of American schooling as the 
teacher-talk model of instruction? The 770 square foot classroom box? The 180-day (American) school 
year? We accept and even welcome critical attention to instructional technology that is seldom applied to 
the implacable regularities of American schooling. That leads to a paradox in which technology from the 
last generation has been proven inadequate and that from the next generation is unproven. With either 
negative data or none, the field is left to those who promptly make the next generation of technology the 
worst enemy of the current generation as in, ’Next year it will be cheaper, faster, smaller or even — more 
constructivist. So let's wait.' 

Our goal should be first, to understand the conditions of pro-social technology use and second to employ 
that understanding for learning improvement. Both require more penetrating analysis than has heretofore 
been the standard. 



B. Pedagogy: The Multiple Sources of Learning. 

The first thing to be understood is that there are many, many sources of learning. Technology needs to be 
disentangled from the other sources. Some are inside and many are outside the school. Parents educate, 
the family educates, the media educates and so on. Children learn from their teachers, from textbooks, 
from homework, from the Channel One TV on the wall and they learn from computers.- 

C. Pedagogy: Learning Outside the School. Thus, the contribution of instructional technology is best 
understood in a context that includes the contribution of all the educators. 

Since James S. Coleman's 1966 analysis, it has been generally acknowledged that about 30% of the 
variation in children's educational achievement comes from their experience in school and 70% comes 
from other experiences, especially their families, the culture they live in (the media, etc.) and their peers. 
Coleman's insight works both ways. Families that support learning, advance their children's educational 
achievement: those that do not or that hinder or disrupt learning, impede their children's educational 
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achievement. In later writing, Coleman called the leverage that families apply to their children's learning, 
"family capital". 

D. Implications for a Research Agenda. There are several implications for an agenda of evaluation 
research about instructional technology. 

1. IT Effects @ Home. We need to account for all the educators— school effects and home effects. How 
many studies are there of the amount of learning at home that is supported by instructional technology? 

2. IT Effects @ School. Inside the school, we need to find ways to measure technology effects separate 
from teacher or textbook effects. How many studies attempt to measure the amounts of these various 
phenomena and associate them with outcomes? 

3. The Effects of Serious Play. Except for the 'leaming-should-hurt' crowd, most educators recall what 
coaches and early childhood educators have never forgotten — play is a child's work. Entertainment is 
correctly pilloried as passive and generally purposeless. But active play is strongly connected to learning 
of all sorts .2 The fact that we do not know much empirically about what and how children are learning 
from technology-delivered serious play applications outlines the limits of our imagination. 

4. Estimating the Critical Mass of IT. We do not have any very good way to answer school board 
questions about, How much is too little? Few school boards would accept a coach assertion that 42 
minutes a week of basketball practice, in groups of 24, will result in 42 offers of college athletic 
scholarships. Why then do we allow policy makers to believe that 42 minutes a week of "Computer" 
from the "The Computer Teacher" in "The Computer Center" will change children's school performance? 

5. IT Dosage. Most current evaluations assume that if a school has bought a site license for Electrified 
Reading: Release 2.0 then the teachers are using it and the children are experiencing (some unknown 
amount) of it. A generation of so-called "implementation research" suggests the fallacy of that 
assumption yet we do not have good measures of children's exposure or even of teachers use of 
programs. We need much more attention to elapsed time, exposure effects, dosage effects. 



III. POLITICS 

"The wireless music box has no imaginable commercial value. Who would pay for a 
message sent to nobody in particular?" Advice to David Samoff in the 1920's. 

Politics is the process through which values are authoritatively distributed for a society. Whatever its 
interpretation in the popular culture, politics has deep implications for the purposes served by 
government action, for example, which children get what quality of schooling and which not? If 
measurement is the essence of science, benefit is the essence of public service, analysis may document 
who benefits from a particular program; politics determines how that benefit will be distributed. 



Consider a choice among three public policy options: What would advance the children's interests more: 
. Higher pay for already employed teachers? 

B. The same amount of money but spent only to hire additional teachers? Or, 

C. The same amount of money spent on instructional technology? 

Or, assume that the policy goal is to "integrate technology into the classroom". The instant consensus is 
that can be done only by more professional development for teachers — more in-service, more released 
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time, more contracts to Teacher Centers or, my own favorite, more subsidies for graduate school tuition. 
We never consider options such as: 

. Giving teachers a computer to take home over the summer (and trusting their professionalism, 
curiosity and commitment) 

B. Putting computers into the classroom and letting the kids explore them and co-teach, co-leam with 
their teachers. 

C. Using technology to teach technology — for example, by producing CD-ROM role-playing 
simulations about what happens in classrooms under different conditions and with different 
teacher choice-consequence paths. 

Neither do we consider making technology so 'transparent' that it does not require training, for example, 
ATMs. For the most part, policy choices are limited by political power and by the conventional 
conceptions of education as schooling and of learning as teaching. They all lead to the same 
labor-intensive conclusions and all are centered on teachers not on learning. We do not, for example ask 
the following question: 

Under what conditions are protein-based teaching systems preferable to digital learning systems? 

It is at least possible that digital systems do things that RLHB's 

• should not bother to do (keep records) 

• do not want to do (drill children) 

• can not do (have infinite patience) or 

• do not do reliably (treat all children as though they can all learn)? 

Or consider "The Learning Odyssey", a complete curriculum for grades 4 though 9, produced iy the 
Agency for Instructional Technology (AIT) that was originally aimed at the home-schooling market. The 
topics include language arts, math, science, history, art, music, technology and personal development. All 
subjects are aligned to state content standards. Teacher comments on student work are available by 
e-mail. Subscription prices are $ 150/month; $350/3 months; $900/9 months; and, $l,100/year. As part of 
the price: AIT will pay for a child to be tested with any standardized test required by a local jurisdiction. 

AIT describes the "Learning Odyssey" as a replacement for school. "..(L)eaming need not be 
school-based... schools must reinvent themselves as institutions with a far greater purpose, or cease to 

exist. "§ 

Or consider another volatile issue, violence and the Internet. Post-Littleton, the concern to minimize the 
sources of violence in children's lives is likely also to generate unintended negative consequences. Our 
understandable attention to the harmful examples of Internet use may cripple wholesome applications of 
the same technology 

We can help children by minimizing violence but also by maximizing good. If we would not ban all 
pharmaceuticals because some are hallucinogens, then we should also differentiate between pro- and 
anti-social applications of telecommunications. Except for a few one-off examples of good video games 
(Tetris, Carmen Sandiego), we have no systematic understanding of the good that can be done through 
learning related games and the Internet. 

In order to encourage more wholesome development of these technologies, we need to understand how 
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thfey work. To continue the pharmaceutical analogy, we need to identify the active ingredients in these 
applications followed by clinical trials to document their effects. We need clinical trials that (1) identify 
and measure the active ingredients of instructional technology and (2) that document the gains associated 
with amounts of their use. 

Whatever the case, analysts, researchers and/or evaluators do not have the right to make deeply political, 
deeply value choices. Analysts are not elected or authorized by any constituency to make official 
decisions. Doctorates are not licenses to usurp state legislatures, local boards of education or even 
superintendents and principals. The role of analysis is to inform decisions, NOT to make them. 

IV. ECONOMICS 

"I think there is a world market for maybe five computers . " 

Thomas Watson, Chairman, IBM, 1943 

A third of a century ago, James Coleman and Lawrence A. Cremin tried to teach us that we needed a 
more generous vision of education than one centered exclusively on schools. In addition, we need a more 
generous vision of progress than that which depends solely on public funds. There are partners who share 
important and child-centered purposes and they are in the private sector, perhaps especially in technology 
in the private sector. 

A. Capitol Decisions and Capital Decisions. Public and private decision-makers are interested in 
answering the same question: What works? They are because both are making investment decisions. "We 
need information to show what works and what doesn't. If we had empirical data, policy-makers would 
be more willing to fund technology and voters would be much more willing to pay. "2 

The interest in 'what works' goes beyond public policy. There is now a category of inquiry called 
"curriculum due diligence". Because potential investors have a right to know about the integrity of what 
is being offered, banks and brokers retain curriculum analysts to document those companies in the 
learning business can actual ly deliver what they claim to sell — learning. Whether the goal is benefit to 
the constituents or return on investment, the interest in efficacy is the same. 

And in both instances there are competing uses for the same funds. Linda Roberts, the director of the 
Office of Educational Technology in die US Department of Education said, "School districts will be. 
called to task for 'What are you doing with your money and what difference does it make?"12 in the 
public sector, the (implicit) questions are: More highways or more schools? More lights on police cars or 
more computers in classrooms? More scholarships for college students or more professional development 
for teachers? The private sector compares buying a magazine to creating software, or starting a chain of 
day care centers to creating an Internet homework helper site. All of those decisions can be illuminated 
by data about outcomes for learning. 

B. The Public Benefits of Private Investment Where has there been more good for schooling? (A)The 
clouds of quarter million dollar grants from foundations and State Departments of Education to fund 
(non-profit) pilot projects and experimental activities or (B) two guys named Jobs and Wozniak trying to 
breadboard a "home computer" in a Cupertino garage? 

We should at least admit that in a capitalist society the engine of innovation— and yes, largely of 
improvement— is the profit motive. The need for parents to be better connected to their children's 
learning is widely acknowledged but who has invested more in creating a curriculum of the home? State 
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departments of education or the Lighispan Partnership? (Hint: Creating the 100+ CD ROMs that support 
reading, language arts and mathematics, grades k through 6 has cost Lightspan $150 million. 

C. The Roles of Government. Recall that the Internet started out as a Defense Agency Research Projects 
Administration experiment to connect weapons labs. The Food and Drug Administration supervises 
clinical trials of privately developed pharmaceuticals on behalf of the public. 

V. FOUR INSTRUCTIONAL TECHNOLOGY ADDITIONS TO THE . 
EXISTING CONCEPTION OF SCHOOLING 

"Computers in the future may weigh no more than 1.5 tons. ""Popular Mechanics", 1949 

A Adding an Education Focus to Our School Focus. There are a lot of educators in every society — the 
TV, newspapers, parents, religious and cultural institutions, video games, sports and the general culture. 
Every time we hear "education" and automatically think "school" we are diminishing the prospects of 
improvement. 

B. Adding a Learning Focus to Our Teaching Focus. The frontal act of instruction, the uncertain 
business of trying to require children to learn particular things is very difficult (ask any teacher). What if 
we re-conceptualized "the teaching of children" as "the facilitation of learning"? Likely, it is more 
possible to arrange learning than it has been to force teaching. 

C. Adding Homes to Schools: adding Parents to Teachers. Schools and homes remain isolated from each 
other. And, despite there centrality in the lives of children, we have never had very good bridges between 
the two. Using Lightspan's Achieve Now! Schools lend children a Sony Playstation (retail cost, $100) as 
a platform for learning-related video games that are launched by the teacher in the classroom but then 
completed by children and their parents at home. In a pre/post and experiemental/control evaluation, the 
children and schools with this home-school-home connection performed better on reading, language arts 
and mathematics than did those without. 

Moving Learning to the Learner. One certain consequence of digital technology is that learning will go 
to the learner. In the earliest times, boys went with their fathers and uncles to observe the hunt; girls went 
with the mothers and aunts to discover which plants were edible. The artists of the cave walls moved 
learning inside. The creation of the common school still required learners to go to the site of learning and 
to dependent on the knowledge masters. Dependency makes learners vulnerable to the political (and 
ethnic and class and gender) prejudices of the masters. 

Digital communications reverses that commerce (with the Internet, learning goes to the learner) and 
dramatically transforms that politics. Digital learning can be "The 4 'Anys' — Any Learning, Any Time, 
Any Place to Any One". The democratizing impacts of that reversal are heartening but only dimly 
perceived. And the consequences for schools and universities, conceived as physical spaces, have not 
begun to be imagined although their consequences are probably captured by the observation about 
technology as a 'train' — you will either be on it or under it. 

VI. FOUR EVALUATION RESEARCH QUESTIONS FROM FOUR 
PERSPECTIVES 

"But what is it good for? " Comment on the microchip from IBM Advanced Computing 
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' systems Division, 1968 

• From the perspective of science, How certain, how unambiguous, how compelling are the data? 

• From the perspective of pedagogy, What implications for practice can be derived from the 
evaluation? 

• From the perspective of politics, What political values are served? 

• From the perspective of economics, What are the cost or financial implications? 

About the Author: Dale Mann is a professor in the Program in Educational Administration at Teachers 
College, Columbia University and Managing Director of Interactive, Inc., a technology development and 
evaluation firm Huntington, New Y ork. 
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